Imagine you show up at your favorite cafe and its credit card terminal is down. What happens? Most likely, you don’t have any cash on you. Instead, you go to the next cafe and order from them instead. We never want this to happen to one of our customers, so that’s why we chose to build our infrastructure on CockroachDB.
At Aurora Payments, we make sure that small and medium sized businesses (SMBs) get paid quickly and with low fees. We integrate with their point of sale (POS) technology and business management software. We learn all the nuances of their business so that we can process and deliver payments exactly the way our customers need them.
We know that all of our effort to deliver exceptional payment solutions would be meaningless if we didn’t have a database that is always on - and always available.
Why Aurora Payments Chose CockroachDB
In some sense, I didn’t really choose the payment solutions industry, it chose me. I was running a cloud software delivery business when my largest client, who just happened to be a payment processor, asked me to join their firm as CTO. That company was acquired by Aurora in 2018, giving us access to capital to build out a powerful technology ecosystem.
When I got into the business, I realized that we had a chance to really make a positive impact on peoples’ lives. I’ve attended well over one hundred industry events and trade shows during my time here, allowing me to connect directly with the merchants that we serve. I have never seen a group of customers who are so emotional about their payment partner, especially when they are bombarded by inbound sales calls from competitors. What I began to understand is not only how critical payment processing is for any business, but how a trusted relationship combined with proprietary technology integration can facilitate transformational change, even in a small family-owned business.
I viewed this as a great technical opportunity and, frankly, a huge responsibility to do right by the people that chose to work with us. Merchants trust us with delivering their money, which can mean the difference between making payroll and not. Our sales partners trust us to deliver their residual income payments, to service their clients, and to provide powerful data in real time.
However, in order to build a next generation payments solution for SMBs, we needed a database bringing strengths that aligned with the priorities of the payment solutions industry and with our vision for the future of Aurora’s service:
• Always on and always available - No customer of ours will ever lose money because their payment solution was unavailable.
• Strong consistency - We won’t ask our customers to deal with a confusing system of eventually consistent transactions. We’ll get it right the first time.
• Kubernetes Compatibility - We were moving to container orchestration and wanted a database that fit well with k8s.
• Horizontal scale - Our intention is to grow across in all of the United States. In my experience you’re never too small to pick a scalable solution early on because the cost of changing up later down the road is greater than just choosing a scalable solution from the start.
High Availability & Scale for Payment Service Workloads
We evaluated other databases beginning with the database we were already using: Postgres. On a single server, I think the performance of Postgres and CockroachDB would be pretty comparable. But the magic of CockroachDB happens when you start to add nodes and distribute the data across those nodes automatically.
The problem with Postgres is that it doesn’t scale well natively in a geo-redundant configuration with an active/passive HA setup. If you try to launch a Postgres instance in NYC and another in Los Angeles, you really have to choose which one is going to be active. The failover process can be somewhat smooth, but it won’t be a sub-second switchover.
At Aurora we want to plan for the times when there’s a site outage that’s completely out of our control - because we know it’s going to happen. And when it does, we need to have a database that will allow us to continue processing payments without skipping a beat.
SQL vs NoSQL in Financial Services
Any database that offers eventual consistency is not an acceptable solution for us. We need an accurate status for every transaction. We can’t run someone’s credit card twice - that’s bad customer service. People have finite credit limits and finite sums of money in their bank accounts - so we can’t risk running duplicate transactions and causing problems for them and our vendors.
In my mind, the only viable solutions for online transaction processing are the relational databases like a traditional Postgres on-prem solution or the distributed SQL databases like CockroachDB - which gives you the same consistency as Postgres, but also gives you the scalability and redundancy that we wanted. The consistency strength of CockroachDB is outstanding, as it guarantees serializable transactions, which is the highest isolation level guaranteed by the SQL standard. This is a new class of database that’s become known as NewSQL or Distributed SQL.
Critical Data Stored in CockroachDB
Currently, CockroachDB serves as our System of Record database (and will eventually become our general-purpose database) and for that reason it stores essentially all of the data required by our customers and our internal users. This includes critically private data which is subject to external audits and vendor risk assessments, all of which is housed in CockroachDB.
Then there’s all the point-of-sale transactional data that you would expect. It gets interesting when you delve into the line level data or behavioral data. For example, if you order the double-tall nonfat latte at 8:00am on Tuesday and you tip 18% - we’re storing all that behavioral data. This has opened up new opportunities for us and our vendor partners.
CockroachDB Helps Unlock New Revenue Streams
Prior to my arrival at Aurora, the company was essentially a reseller of other companies' payment solutions. Now that we’ve begun building our own payment ecosystem, we have access to a wealth of transaction data. And data, as you know, is powerful.
Now, in addition to our payment solution business, we are also building a customer loyalty business. Marketing and payment services are traditionally thought of as distinct from one another. But they absolutely fit well together. We can use the drink order, drink time, tip amount, and other behavioral data to help our customers send the right promotional email or text, at the right time, to the right people. And we can create a more seamless experience for the customer in the cafe by pre-loading their preferred tip amount and email preferences.
It’s worth noting that having all this data at our fingertips also makes it much easier for us to understand our cash flow. We make money when our customers are processing payments. Conversely, if a customer is forced to close, or has a bad month, we feel the effects as well.
Real time visibility into the data, particularly during the challenges of the past year, has been extremely valuable to us.
Payment Solutions Application Architecture with CockroachDB
A simple overview of our architecture and tech stack looks like this:
• Front end is React
• Back end is .Net Core C#
• Database: CockroachDB (still migrating some workloads off of Postgres)
- Debian for backend and front-end webservers
- A combination of load balancers managed by Google and based on HAProxy
• Kubernetes
• Hybrid-Cloud: Google Cloud & two co-location facilities (We’ll never be 100% public cloud or 100% private). We have private interconnection capacity between each of our datacenter facilities and Google Cloud, which means our internal traffic never traverses the public Internet.
How CockroachDB gives Aurora a Competitive Advantage
My observation of our competitors is that most of them are running on legacy database systems like MySQL, standalone Postgres, or Microsoft SQL Server. One of our competitors that is many times our size, has built its service, reporting, and analytics platform on a SQL Server backend. We’ve noticed that there are times of the day when it can be almost completely unusable. This leads to a poor experience for the competitor’s sales partners, merchants, and employees. We are also aware of another player in the industry who has attempted to solve for the availability issue by building its database on a public cloud’s native mySQL offering, only to discover during one particularly disruptive cloud outage that not only is their cross-regional database not an immediate failover, but they are now heavily vendor-locked-in and are unable to leverage a multi-cloud strategy.
By using CockroachDB, we’ve removed the database from being a bottleneck for performance or availability. The database should not be a bottleneck in the first place, but legacy systems were not built for modern applications. Choosing CockroachDB makes us more advanced than the vast majority of the financial services industry, even firms many times our size. Everyone is capable of spinning up thousands of front-end servers (and paying lots for it) or processes to deal with requests - but not very many people consider fixing the core problem: the database itself. They just use what they know - and deal with the bottlenecks.
There’s a productivity advantage as well. Because CockroachDB is wire-compatible with Postgres, it speaks SQL and makes the migration process fairly easy. CockroachDB eliminates a lot of the manual labor that I’m accustomed to. The ability to spin up a non-production cluster with just a few lines of code is really impressive. Traditionally, that just isn’t how databases work. There’s supposed to be some kind of challenge or struggle that you have to conquer manually!
My development team members had to acquaint themselves with the architecture of CockroachDB to fully understand its potential. Once they saw the ease of use and power of the database, they got comfortable with it. When we start to add nodes and distribute data, then we’ll really reap the competitive benefits of choosing CockroachDB.
What’s next for Aurora Solutions & CockroachDB
First, we need to get all of our transactional workloads off of Postgres and onto CockroachDB, and they are in the process of being moved now. We also will be migrating our initial bare metal clusters of CockroachDB fully onto Kubernetes because we were impressed with how easy it is to manage data that way. The next step will be to continue to build the RISE family of payment processing products, which are all workloads that will run on CockroachDB. At this point, CockroachDB will become our general-purpose database. To be frank, I’m very excited about it because I think it will help our customers a great deal and, to circle back to the beginning of this story, that’s what makes this work meaningful to me. We can make a positive impact on the lives of small and medium size business owners by not only saving them money, but also offering a trusted partnership for delivering payment solutions. CockroachDB helps us do that.
